Skip to content

storage: Add a FAILOVER option for PostgreSQL sources - #40686

Merged
peterdukelarsen merged 7 commits into
MaterializeInc:mainfrom
peterdukelarsen:peterlarsen/cs-752-pg-slot-failover
Nov 13, 2026
Merged

storage: Add a FAILOVER option for PostgreSQL sources#40686
peterdukelarsen merged 7 commits into
MaterializeInc:mainfrom
peterdukelarsen:peterlarsen/cs-752-pg-slot-failover

Conversation

@peterdukelarsen

@peterdukelarsen peterdukelarsen commented Nov 12, 2026 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Contributor

Motivation

PostgreSQL 17+ only synchronizes a logical replication slot to standbys if it was created with FAILOVER, which PostgreSQL sources could not request (part of CS-752).

Description

Adds a FAILOVER option to CREATE SOURCE ... FROM POSTGRES, gated by the feature flag enable_postgres_source_failover (default off), that creates the source's slot with FAILOVER and is rejected unless the upstream is a PostgreSQL 17+ primary.

Verification

A new replication-slot-failover workflow in test/pg-cdc checks the feature flag gate and asserts pg_replication_slots.failover for a source created with the option, and a parser test covers the syntax.

🤖 Generated with Claude Code

peterdukelarsen and others added 2 commits November 12, 2026 15:18
@peterdukelarsen @claude
storage: Add a flag to create PostgreSQL replication slots with failover
e5bb182 
PostgreSQL 17+ synchronizes a logical replication slot to standbys only
if the slot was created with FAILOVER. A source's slot was created
without it, so the slot did not survive an upstream failover.

The new dyncfg `pg_source_replication_slot_failover` (default off) makes
a source create its slot with FAILOVER. It is replica-scoped, so it can
be targeted at single clusters. Servers that cannot create such a slot
(older than PostgreSQL 17, or a standby) get a plain slot as before.
Existing slots are not altered.

Part of CS-752

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@peterdukelarsen @claude
human review
43c87b8 
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@peterdukelarsen
peterdukelarsen force-pushed the peterlarsen/cs-752-pg-slot-failover branch from 66b89f8 to 43c87b8 Compare November 12, 2026 20:21
@peterdukelarsen @claude
Replace the dyncfg with a FAILOVER source option behind a feature flag
e5eb133 
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@peterdukelarsen peterdukelarsen changed the title storage: Add a flag to create PostgreSQL replication slots with failover storage: Add a FAILOVER option for PostgreSQL sources Nov 12, 2026
peterdukelarsen and others added 4 commits November 12, 2026 15:46
@peterdukelarsen @claude
Gate FAILOVER on its presence and test all three forms
1e834d4 
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@peterdukelarsen @claude
Set REPLICA IDENTITY FULL on the failover test table
655449f 
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@peterdukelarsen @claude
Give each failover test source a table so it creates its slot
c49cfec 
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@peterdukelarsen @claude
Regenerate the system parameter catalog for the failover flag
4efab98 
The checked-in src/sql/system-parameters.json lacked
enable_postgres_source_failover, which failed the
checked_in_catalog_is_current test in mz-system-parameter-catalog.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@peterdukelarsen
peterdukelarsen marked this pull request as ready for review November 13, 2026 15:08
@peterdukelarsen
peterdukelarsen requested review from a team as code owners November 13, 2026 15:08
patrickwwbutler
patrickwwbutler approved these changes Nov 13, 2026
View reviewed changes

@patrickwwbutler patrickwwbutler left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, all follows normal patterns for these things!

@peterdukelarsen
peterdukelarsen enabled auto-merge (squash) November 13, 2026 15:15
@martykulma

martykulma commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Instead of introducing a new flag, should we enabled this behavior only if pg_source_validate_timeline is enabled? Unless they're both enabled, a failover would still cause a timeline id switch and brick the source.

@patrickwwbutler

patrickwwbutler commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Instead of introducing a new flag, should we enabled this behavior only if pg_source_validate_timeline is enabled? Unless they're both enabled, a failover would still cause a timeline id switch and brick the source.

Is there a scenario where someone would want pg_source_validate_timeline but not FAILOVER? If not, then that does make sense, would be nice to keep total number of flags down, it can get cumbersome

@peterdukelarsen
peterdukelarsen merged commit 434500c into MaterializeInc:main Nov 13, 2026
89 checks passed
@martykulma

martykulma commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Is there a scenario where someone would want pg_source_validate_timeline but not FAILOVER? If not, then that does make sense, would be nice to keep total number of flags down, it can get cumbersome

Setting this flag on the slot only makes it eligible for synchronization, the user still has to set the PG server flag to enable the functionality.

The scenario where a user wants to enable sync_replication_slots and disable timeline validation, but not allow MZ to set the failover flag on the replication slot doesn't make a lot of sense to me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@patrickwwbutler patrickwwbutler patrickwwbutler approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@peterdukelarsen @martykulma @patrickwwbutler